Skip to content

feat(sandbox): add GCE metadata emulator for Google Cloud#1763

Merged
maxamillion merged 5 commits into
NVIDIA:mainfrom
p5:dev/robertsturla/gce-metadata-emulator
Jun 23, 2026
Merged

feat(sandbox): add GCE metadata emulator for Google Cloud#1763
maxamillion merged 5 commits into
NVIDIA:mainfrom
p5:dev/robertsturla/gce-metadata-emulator

Conversation

@p5

@p5 p5 commented Jun 4, 2026

Copy link
Copy Markdown
Contributor

Summary

Right now you can't use Google Cloud APIs (Vertex AI, Cloud Storage, BigQuery, Drive, Maps, etc.) from inside a sandbox. GCP SDKs expect a metadata server to be running and query it to get tokens - but there's no metadata server in the sandbox, so they fail before any API call is even attempted.

Go's metadata client makes this worse. It dials the metadata IP directly over TCP, bypassing HTTP_PROXY entirely, so the sandbox proxy never even sees the request. Additionally, there's no way to override this from within the SDK config.

This PR adds a google-cloud provider type and a GCE metadata emulator running on loopback (127.0.0.1:8174) inside the sandbox network namespace. GCP SDKs find it via GCE_METADATA_HOST, get credential placeholders back, and include those in their API calls. The proxy resolves placeholders to real tokens at egress. The sandbox process never holds a real credential.

Related Issue

Closes #1706

Changes

  • Add google_cloud module with shared constants and loopback address
  • Add google_cloud_metadata module implementing GCE metadata API
  • Add metadata_server module with MetadataHandler trait for provider- agnostic loopback server lifecycle
  • Add child_env_resolved() and gcp_token_response() for GCP-aware credential state
  • Bind via std::thread::spawn + setns (not spawn_blocking) to avoid tokio thread pool namespace contamination
  • Start metadata server before SSH handler to ensure consistent env on bind failure
  • Add google-cloud.yaml provider profile and credentials documentation

Testing

  • mise run pre-commit passes
  • Unit tests added/updated
  • E2E tests added/updated (if applicable)

Checklist

  • Follows Conventional Commits
  • Commits are signed off (DCO)
  • Architecture docs updated (if applicable)

@copy-pr-bot

copy-pr-bot Bot commented Jun 4, 2026

Copy link
Copy Markdown

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

@github-actions

github-actions Bot commented Jun 4, 2026

Copy link
Copy Markdown

All contributors have signed the DCO ✍️ ✅
Posted by the DCO Assistant Lite bot.

@p5

p5 commented Jun 4, 2026

Copy link
Copy Markdown
Contributor Author

I have read the DCO document and I hereby sign the DCO.

@p5 p5 force-pushed the dev/robertsturla/gce-metadata-emulator branch 5 times, most recently from cb2e254 to 28d626a Compare June 4, 2026 21:47
@maxamillion maxamillion added the test:e2e Requires end-to-end coverage label Jun 4, 2026
@github-actions

github-actions Bot commented Jun 4, 2026

Copy link
Copy Markdown

Label test:e2e applied, but pull-request/1763 is at {"messa while the PR head is 28d626a. A maintainer needs to comment /ok to test 28d626a9e068293469bbeb1661bdb77e973112df to refresh the mirror. Once the mirror catches up, re-run Branch E2E Checks from the Actions tab.

@maxamillion

Copy link
Copy Markdown
Collaborator

/ok to test 28d626a

@p5 p5 force-pushed the dev/robertsturla/gce-metadata-emulator branch from 28d626a to abde14b Compare June 4, 2026 22:16
@p5

p5 commented Jun 4, 2026

Copy link
Copy Markdown
Contributor Author

Force pushed to fix the lint issues. "mise run pre-commit" now succeeds locally again.

Previous E2E tests ongoing here - https://github.com/NVIDIA/OpenShell/actions/runs/26982657920


Edit:
rust-docker E2E tests failed. It looks like a flake, especially given the same tests for podman are passing.

@maxamillion

Copy link
Copy Markdown
Collaborator

This is great! One nit pick though, I think GCE_METADATA_IP should include the emulator port. The google-auth python library builds its metadata ping URL directly from GCE_METADATA_IP, so 127.0.0.1 becomes http://127.0.0.1 on port 80 but the shim listens on 127.0.0.1:8174 ... from an initial check, it looks like the other language libraries for google auth use GCE_METADATA_HOST

Tested with:

$ env UV_CACHE_DIR=/tmp/uv-cache uv run --with google-auth --with requests python /tmp/verify_gce_metadata_ip.py
ping_127_no_port False
ping_127_with_port True

here's the contents of /tmp/verify_gce_metadata_ip.py:

import os
import threading
import http.server
import socketserver

import google.auth.compute_engine._metadata as metadata
from google.auth.transport.requests import Request


class MetadataHandler(http.server.BaseHTTPRequestHandler):
    def do_GET(self):
        self.send_response(200)
        self.send_header("Metadata-Flavor", "Google")
        self.end_headers()

    def log_message(self, *args):
        pass


def main():
    server = socketserver.TCPServer(("127.0.0.1", 8174), MetadataHandler)
    thread = threading.Thread(target=server.serve_forever, daemon=True)
    thread.start()

    try:
        request = Request()

        os.environ["GCE_METADATA_IP"] = "127.0.0.1"
        print("ping_127_no_port", metadata.ping(request, timeout=1, retry_count=1))

        os.environ["GCE_METADATA_IP"] = "127.0.0.1:8174"
        print("ping_127_with_port", metadata.ping(request, timeout=1, retry_count=1))
    finally:
        server.shutdown()
        server.server_close()


if __name__ == "__main__":
    main()

@p5 p5 force-pushed the dev/robertsturla/gce-metadata-emulator branch from abde14b to 009f28a Compare June 4, 2026 22:51
@p5

p5 commented Jun 4, 2026

Copy link
Copy Markdown
Contributor Author

Awesome spot!
You're correct - the variable should include the port.

Ran through your reproducer and confirmed it now works as expected.

@p5 p5 marked this pull request as ready for review June 5, 2026 11:20
@p5 p5 requested review from a team, derekwaynecarr, maxamillion and mrunalp as code owners June 5, 2026 11:20
@p5 p5 force-pushed the dev/robertsturla/gce-metadata-emulator branch 2 times, most recently from 2d776a7 to aada3da Compare June 5, 2026 17:48
@cgwalters

Copy link
Copy Markdown
Contributor

One thing I'd like to consider here is OpenShell including something like https://github.com/LobsterTrap/llmproxy by default - i.e. an inference endpoint that always appears to be OpenResponses compatible to inner tooling. Wouldn't work with Claude Code (AFAIK without hacks) but it'd be nice to just entirely remove needing to handle inference provider auth at all for all the tools that can speak OpenResponses.

It's a heavier hammer here though.

Of course it's worth noting that many non-local deployments will probably end up wanting some kind of proxy anyways to handle observability etc. There's various existing more heavyweight things in that space.

@stbenjam

stbenjam commented Jun 6, 2026

Copy link
Copy Markdown

Thanks, these changes work for me standalone following Adam's instructions.

Will vertex provider stay or get stripped out? It might be confusing if it exists but doesn't work with CC.

@TaylorMutch

Copy link
Copy Markdown
Collaborator

/ok to test aada3da

@p5

p5 commented Jun 8, 2026

Copy link
Copy Markdown
Contributor Author

Will vertex provider stay or get stripped out? It might be confusing if it exists but doesn't work with CC.

It's out of scope for this PR to deprecate the original vertex provider, nor does this PR label itself as an inference provider - even though it can be used in that way.

There are discussions ongoing as to whether the original should be deprecated. There are pros and cons for each.
The Google Cloud provider (this PR) gives OpenShell less visibility into inference as requests do not go through inference.local, but does provide more seamless use of Google SDKs in coding agents (#1752 wouldn't have been needed with this).

if it exists but doesn't work with CC.

Was this a regression from my PR? CC should remain working with the original provider following the merge of #1752.

@stbenjam

stbenjam commented Jun 8, 2026

Copy link
Copy Markdown

This PR gives the foundation to let inference.local work without needing the complicated L7 translation layer, and there are lots of benefits to that. I agree its out of scope to do more to the vertex provider here.

if it exists but doesn't work with CC.
Was this a regression from my PR? CC should remain working with the original provider following the merge of #1752.

No it wasn't clear to me that 1752 was going to land.

This PR works well for me, thanks.

@p5 p5 force-pushed the dev/robertsturla/gce-metadata-emulator branch from aada3da to 2b8e14e Compare June 11, 2026 09:25
p5 added 2 commits June 22, 2026 21:13
Single source of truth for GCP naming: env var aliases, provider config
keys, token search order, and Vertex-specific env vars. Consumed by
openshell-server, openshell-providers, and openshell-sandbox.

- Add google_cloud.rs with metadata emulator host and loopback address
- Define PROJECT_ID, REGION, and SERVICE_ACCOUNT_EMAIL env var aliases
- Add provider config key constants for gcp provider implementations
- Define TOKEN_ENV_KEYS search order (SA token takes priority over ADC)
- Add Vertex-specific env vars for Goose and Claude Code SDK integration
- Add STATIC_CONFIG_KEYS as union of all alias arrays for env resolution
- Export module via openshell-core lib.rs

Signed-off-by: Robert Sturla <rsturla@redhat.com>
Add GoogleCloudProvider and VertexProvider implementing inject_env to
project GCP config (project ID, region, SA email, metadata host) into
sandbox environment variables. Replace the inline Vertex AI env
injection in the server with the registry-based inject_env dispatch.

Also adds the google-cloud.yaml provider profile with SA JWT and ADC
OAuth2 credential refresh flows.

Signed-off-by: Robert Sturla <rsturla@redhat.com>
@p5 p5 force-pushed the dev/robertsturla/gce-metadata-emulator branch 4 times, most recently from 7a1c143 to 5d6ba08 Compare June 22, 2026 21:10
@maxamillion

Copy link
Copy Markdown
Collaborator

/ok to test 5d6ba08

maxamillion
maxamillion previously approved these changes Jun 22, 2026

@maxamillion maxamillion left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is great, thank you! I'm a big fan of this approach. 👍

Comment thread crates/openshell-core/src/provider_credentials.rs Outdated
p5 added 3 commits June 23, 2026 17:45
Add a loopback HTTP server on 127.0.0.1:8174 inside the sandbox
network namespace that emulates the GCE instance metadata API.
GCP client SDKs discover it via GCE_METADATA_HOST and obtain
credential placeholders that the proxy resolves to real tokens
at egress.

Add metadata_server module with MetadataHandler trait and
netns-aware TCP binding via std::thread (not spawn_blocking)
to avoid tokio pool namespace contamination
Add google_cloud_metadata module implementing the GCE metadata
API subset (token, project-id, email, scopes, service-accounts)
Add child_env_resolved() and gcp_token_response() to
ProviderCredentialState for GCP-aware credential projection
Wire metadata server into sandbox lifecycle before SSH handler
Collapse multi-line HTTP response format string into single line

Signed-off-by: Robert Sturla <rsturla@redhat.com>
Document the google-cloud provider setup for ADC and service account
flows, injected environment variables, metadata emulator behavior, and
network policy configuration for GCP APIs.

Signed-off-by: Robert Sturla <rsturla@redhat.com>
Widen --from-gcloud-adc to accept google-cloud providers. The ADC
credential key is derived from the provider profile rather than
hardcoded per type, so future GCP provider types get ADC support by
declaring the right refresh metadata in their profile YAML.

Add ProviderTypeProfile::adc_credential() to find the ADC-compatible
credential from a profile's refresh metadata. Remove unused
VERTEX_AI_ADC_TOKEN_KEY and GCP_ADC_TOKEN_KEY constants.

Signed-off-by: Robert Sturla <rsturla@redhat.com>
@p5 p5 force-pushed the dev/robertsturla/gce-metadata-emulator branch from 5d6ba08 to fa02fe2 Compare June 23, 2026 16:46
@maxamillion

Copy link
Copy Markdown
Collaborator

/ok to test fa02fe2

@maxamillion

Copy link
Copy Markdown
Collaborator

rerunning failed jobs, it looks like CI timed out trying to talk to ghcr.io

@maxamillion maxamillion merged commit 48545cf into NVIDIA:main Jun 23, 2026
70 of 72 checks passed
rhuss added a commit to cc-deck/cc-deck that referenced this pull request Jun 28, 2026
* Update brainstorm 072: resolve open questions, choose skill-first approach

All 5 open questions decided: dual-phase asset verification, refresh-time
snippet validation, native+npm fallback for Claude Code, post_install
dry-run at capture, single spec for all 13 changes.

Assisted-By: 🤖 Claude Code

* Update brainstorm overview with 072 entry

Assisted-By: 🤖 Claude Code

* WIP: save before ship (submodule update)

Assisted-By: 🤖 Claude Code

* [Spec Kit] Add specification

* [Spec Kit] Add implementation plan

* [Spec Kit] Add tasks

* feat: implement 13 build skill fixes to eliminate build iterations

Build skill (cc-deck.build.md):
- Add USER root after header for OpenShell builds (non-root base images)
- Add GitHub release asset verification protocol (A2 + C2)
- Change snippet handling to allow modification of broken download commands
- Add Claude Code npm fallback on OOM (exit 137)
- Add exact jq merge command for settings.json (A2 + C2)
- Add post_install sandboxing protocol (A2 + C2)
- Add marketplace setup before plugin installs (A2)
- Add shell config dependency scanning to C2 base image probing
- Add OpenShell base image documentation to Key Rules

Capture skill (cc-deck.capture.md):
- Add compinit preamble rule when stripping plugin managers (Step 5c)
- Add shell config dependency scanning for implicit tools (Step 5c)
- Add fzf GitHub release detection for version compatibility (Step 5c)
- Add GitHub release asset verification at capture time (Step 11)
- Add post_install dry-run validation at capture time (Step 11)
- Add build refresh URL verification note (Key Rules)

Templates:
- Guard starship init with TERM!=dumb check (05-shell-finalize.tmpl)
- Fix cache dir ownership to parent directory (03-mandatory-stack.tmpl)
- Add Claude Code npm fallback (03-mandatory-stack.tmpl)
- Add marketplace setup (03-mandatory-stack.tmpl)

Resolves: brainstorm/072-build-skill-iteration-reduction.md

Assisted-By: 🤖 Claude Code

* Mark all implementation tasks complete

Assisted-By: 🤖 Claude Code

* fix: use probe results instead of hardcoded tool list in Key Rules

The OpenShell base image tool list is dynamic. Rely on the base image
probe and shell config scanning rather than listing specific missing
tools that will drift as the base image evolves.

Assisted-By: 🤖 Claude Code

* fix: use cp via Bash for config file copies in capture skill

The Write tool requires reading a file before overwriting it, which
fails silently when re-running capture on an existing setup directory.
Using cp via Bash ensures config files are always updated with current
local versions.

Assisted-By: 🤖 Claude Code

* Add brainstorm 073: OpenShell SSH-to-HTTPS git clone conversion

SSH git clones fail in OpenShell sandboxes because DNS queries (UDP)
bypass the HTTP CONNECT proxy. HTTPS works because DNS resolves inside
the CONNECT tunnel. Fix: convert SSH URLs to HTTPS + git insteadOf.

Assisted-By: 🤖 Claude Code

* [Spec Kit] Add specification

* [Spec Kit] Add implementation plan

* feat: convert SSH git URLs to HTTPS for OpenShell sandboxes

OpenShell's HTTP CONNECT proxy cannot resolve DNS for SSH connections
(UDP bypasses proxy). Add sshToHTTPS() conversion in buildCloneCommand()
for OpenShell workspaces, plus git insteadOf config in OpenShell images
for submodules and other in-sandbox git operations.

Assisted-By: 🤖 Claude Code

* fix: patch lsd/bat configs for sandbox color compatibility

lsd's auto color detection fails inside Zellij panes (isatty returns
false). The capture skill now patches copied config files to force
color and icon output, since sandboxes always run inside a multiplexer.

Assisted-By: 🤖 Claude Code

* fix: always overwrite context files from setup/config during build

Stale config copies in openshell/context/ from previous builds caused
lsd color settings to not take effect. The build skill now explicitly
requires overwriting existing context files.

Assisted-By: 🤖 Claude Code

* fix: purge stale config files from context dir on every build

Delete non-binary files from openshell/context/ before copying fresh
configs from setup/config/. Preserves cc-deck-linux-* binaries which
are expensive to re-download. Prevents config staleness permanently.

Assisted-By: 🤖 Claude Code

* Add brainstorm 074: OpenShell sandbox resource limits

OpenShell defaults to 2 vCPU / 2 GB RAM, insufficient for Rust/Java
builds. Add --cpu and --memory to ws new, plus manifest defaults with
capture-time detection for heavy projects.

Assisted-By: 🤖 Claude Code

* fix: add compdef overrides for aliased commands (ls/cat/vim)

When ls is aliased to lsd, zsh still uses _ls for completion, causing
stem duplication. Add compdef _files overrides after alias definitions
so tab completion uses simple file completion instead of the original
command's completer.

Assisted-By: 🤖 Claude Code

* fix: set UTF-8 locale in sandbox shells for correct tab completion

The OpenShell base image defaults to POSIX locale (LC_CTYPE=POSIX).
With POSIX locale, zsh miscalculates the display width of multi-byte
characters in the starship prompt, causing a 2-character cursor offset
on every tab completion. Setting LANG=C.UTF-8 fixes the width
calculation.

Assisted-By: 🤖 Claude Code

* revert: remove alias compdef overrides (root cause was POSIX locale)

The stem duplication was caused by LC_CTYPE=POSIX miscounting
multi-byte prompt characters, not by _ls completing for the lsd alias.
The UTF-8 locale fix in 05-shell-finalize.tmpl resolves the actual
root cause.

Assisted-By: 🤖 Claude Code

* docs: document UTF-8 locale requirement in build and capture skills

Add explanations for why LANG=C.UTF-8 is needed (POSIX locale breaks
zsh tab completion with multi-byte prompt chars) to Key Rules and
capture Step 5c. The template handles it automatically, but the skills
document the requirement for manual config scenarios.

Assisted-By: 🤖 Claude Code

* fix: enforce /usr/local/bin for GitHub release tool installs

OpenShell's filesystem policy only grants read access to /usr, /lib,
and /sandbox. Binaries installed to /opt or left in extracted
subdirectories fail with permission denied at runtime. Both build and
capture skills now document that install_path must be /usr/local/bin/.

Assisted-By: 🤖 Claude Code

* fix: initialize LS_COLORS via dircolors for rich file-type coloring

Without LS_COLORS, lsd only colors directories. The OpenShell base
image has dircolors available but it is not invoked during shell init.
Adding eval dircolors to bashrc/zshrc provides file extension colors
for archives, executables, images, etc.

Assisted-By: 🤖 Claude Code

* fix: preserve configured targets when init --force overwrites manifest

Previously, init --force replaced the manifest with a fresh template
where all targets are commented out. This caused build refresh to skip
snippet regeneration for previously configured targets (e.g., openshell).
Now reads the existing manifest's targets before overwriting and carries
them forward into the new template.

Assisted-By: 🤖 Claude Code

* fix: init --force preserves existing manifest (user data from capture)

Previously --force overwrote build.yaml with a fresh template, wiping
all tools, plugins, settings, and target configuration. Now --force
only re-extracts embedded artifacts (skills, templates, snippets,
scripts) and leaves the manifest untouched. Use /cc-deck.capture
--fresh to reset the manifest if needed.

Assisted-By: 🤖 Claude Code

* fix: hardcode LS_COLORS instead of relying on dircolors database

Ubuntu 24.04's dircolors ships without a database file, producing
empty LS_COLORS. Embed a standard color scheme directly covering
archives, executables, images, media, and common source file types.

Assisted-By: 🤖 Claude Code

* fix: revert hardcoded LS_COLORS, strengthen /usr/local/bin rule

Revert LS_COLORS hardcoding (not maintainable). Strengthen the install
path rule to explicitly forbid symlinks from /opt, and document the
pattern for tools with runtime data directories (copy binary to
/usr/local/bin, runtime to /usr/share, set env var).

Assisted-By: 🤖 Claude Code

* Add brainstorm 075: OpenShell native Vertex provider

Replace cc-deck's homegrown Vertex AI credential handling for OpenShell
workspaces with OpenShell's native google-cloud provider type (GCE
metadata emulator, merged in NVIDIA/OpenShell#1763).

Assisted-By: 🤖 Claude Code

* Update brainstorm overview

Assisted-By: 🤖 Claude Code

* WIP: save before ship

Assisted-By: 🤖 Claude Code

* [Spec Kit] Add specification

* [Spec Kit] Add implementation plan

* [Spec Kit] Add tasks

* feat: use OpenShell native google-cloud provider for Vertex AI

Replace cc-deck's homegrown Vertex credential handling for OpenShell
workspaces with OpenShell's native google-cloud provider type (GCE
metadata emulator, NVIDIA/OpenShell#1763).

Changes:
- Update claude-vertex profile to type "google-cloud" (was "claude")
- Remove FileVar and Endpoints from claude-vertex (OpenShell handles it)
- Remove standalone "vertex" provider profile (dead code)
- Rewrite ResolveCredentials claude-vertex path: creates google-cloud
  provider with --from-gcloud-adc and --config instead of SkipProvider
  with file upload
- Update CreateProvider to use --from-gcloud-adc and --config flags
  for google-cloud provider type
- Skip file upload for google-cloud providers in ws/openshell.go
- Keep env var injection for CLAUDE_CODE_USE_VERTEX=1 and companions
- Non-OpenShell workspace types unchanged

Assisted-By: 🤖 Claude Code

* docs: update README for native OpenShell Vertex provider

Assisted-By: 🤖 Claude Code

* review: add code review for 073-openshell-native-vertex

Compliance: 9/9 FRs satisfied. Deep review: 0 Critical, 0 Important.
Gate: PASS.

Assisted-By: 🤖 Claude Code

* fix: handle google-cloud provider type in UpdateProvider

UpdateProvider was passing --from-existing + --credential for
google-cloud providers, which the OpenShell CLI rejects. Use
--from-existing + --config instead, matching CreateProvider behavior.

Assisted-By: 🤖 Claude Code

* fix: delete+recreate google-cloud providers instead of update

The OpenShell CLI's provider update command doesn't support
--from-gcloud-adc, so EnsureProvider now deletes and recreates
google-cloud providers when they already exist.

Assisted-By: 🤖 Claude Code

* Update brainstorm 049: decide on full gRPC replacement

Revisited after Vertex provider migration exposed three runtime CLI
flag incompatibilities. Research shows CLI is not needed for SSH or
file transfer (corrects original assumption). Decision: full gRPC
client replacing CLI wrapping entirely.

Assisted-By: 🤖 Claude Code

* Update brainstorm overview

Assisted-By: 🤖 Claude Code

* WIP: save before ship - add virtual sort fix brainstorm

Assisted-By: 🤖 Claude Code

* fix: replace physical tab sort with virtual display-only sort (#4)

The sidebar session sort (S key in navigate mode) used async MoveTab API
calls to physically reorder Zellij tabs. This caused a race condition
where TabUpdate events overwrote the proactive state updates, reverting
the sort on every sidebar refresh.

Replace with a virtual display-only sort: the controller toggles a
sort_active flag, and the render broadcast sorts sessions by activity
tier (Active > Inactive > Paused) without moving Zellij tabs. The sort
is instant, race-free, and toggles on/off with repeated S presses.

Additional fixes included in this change:
- Fix credential dedup when claude-vertex type diverges from base
- Fix pre-existing Go test failures (containerfile, pipe channel, integration)
- Fix clippy 1.96 warnings (checked_div, sort_by_key, const thread_local)
- Add zip format handling to build asset verification
- Amend constitution to v1.3 (command files are executable code)

353 Rust tests pass, 0 clippy warnings.

Assisted-By: 🤖 Claude Code
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

test:e2e Requires end-to-end coverage

Projects

None yet

Development

Successfully merging this pull request may close these issues.

feat: emulate GCE metadata server for Google SDK access in sandboxes

6 participants